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METHOD AND APPARATUS FOR ORGANIZING DIGITAL MEDIA 
BASED ON FACE RECOGNITION 

CROSS-REFERENCE TO RELATED APPLICATION 
[0001] This non-provisional application claims priority under 35 U.S.C. § 1 19(e) of U.S. 

Provisional Application No. 60/519,639 filed November 14, 2003, the entire contents of which 
are hereby incorporated by reference. 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

[0002] This invention relates to a computer-based technique for organizing media, and 

more particularly to a method and apparatus for organizing digital media based on face 
recognition. 

2. Description of the Related Art 

[0003] Digital cameras have gained popularity in recent in years, in part due to the 

flexibility offered by electronic image storage. By storing digital photos on a personal computer, 
laptop, a network accessible server, etc., users are able to organize, edit, and share their images. 
Also, as compared to film-based photography, the user of a digital camera can typically assess 
the quality of a photo immediately, without spending time and money on a set of prints 
containing at least some low quality images. In view of the widespread transition from analog to 
digital photos, service providers and vendors have introduced a variety of products, web-based 
services, and software tools, including processing software tools for editing digital photos and 
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services for remotely ordering prints and other products (e.g., holiday cards) using the user's 
picture file(s). File browsing and media viewing software typically allows users to organize and 
view their photos in electronic Albums, for example organized in folders based on dates (e.g., 
Oct. 2003) and events (e.g., Ski Trip - Jan. 2002). Considering that the average user has 
hundreds of photos, such tools offer a convenient way to organize and retrieve photos. 
[0004] Despite these existing products and services, the inventors of this application has 

found that the need exists for more-advanced techniques and software tools for conveniently and 
quickly organizing digital photos. 

SUMMARY OF THE INVENTION 
[0005] In one aspect, the present invention is directed to a computer-based method and 

apparatus for organizing digital media, particularly digital photos, using recognition techniques. 
According to a first aspect of the present invention, a computer-based method for organizing 
digital photos comprises: extracting objects of interest from a plurality of digital photos; 
cropping the plurality of digital photos to generate images of isolated objects of interest; 
applying a recognition algorithm to determine the similarity of isolated objects with a reference; 
displaying a plurality of objects of interest arranged as a function of the determined similarity; 
and receiving user input to associate the objects and corresponding digital photos with a 
particular classification. 

[0006] According to a second aspect of the present invention, an apparatus for organizing 

digital photos comprises: an object detecting unit for detecting objects of interest in a plurality of 
digital photos and cropping the plurality of digital photos to generate images of isolated objects 
of interest; a recognition unit for applying a recognition algorithm to determine the similarity of 
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isolated objects with a reference; a display output for displaying a plurality of objects arranged as 
a function of the determined similarity; and a user input for receiving user input to associate the 
objects and corresponding digital photos with a particular classification. 

[0007] In accordance with one embodiment of the present invention, the operations of 

applying the recognition algorithm and displaying image objects as a function of determined 
similarity are repeated as objects are classified. In one embodiment, the objects are faces. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0008] Further aspects and advantages of the present invention will become apparent 

upon reading the following detailed description and with reference to the appended drawings, in 
which: 

[0009] FIG. 1 is a block diagram of a system for organizing digital media according to an 

embodiment of the present invention; 

[0010] FIG. 2 is a block diagram illustrating in more detail aspects of the media 

organizing and processing unit of the system illustrated in FIG. 1 according to an embodiment of 
the present invention; 

[0011] FIG. 3 is an exemplary graphical user interface display screen associated with 

media organizing and browsing according to an embodiment of the present invention; 
[0012] FIG. 4 is a flow diagram illustrating operations performed for recognition-based 

organization of digital photos in accordance with an embodiment of the present invention; 
[0013] FIG. 5 is an exemplary graphical user interface display screen associated with 

importing images for face recognition-based organizing of digital photos in accordance with an 
embodiment of the present invention; 
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[0014] FIG. 6 is an exemplary graphical user interface display screen associated with 

displaying face images sorted by similarity to model face images to assist the user in organizing 
digital photos; and 

[0015] FIGs. 7A-7C illustrate an exemplary digital photo presentation using the results of 

face recognition in accordance with an embodiment of the present invention. 

DETAILED DESCRIPTION 
[0016] Embodiments of the present invention are more specifically set forth in the 

following description with reference to the appended figures. Generally, the present invention is 
directed to a computer-based method and system for organizing media, and more particularly to 
a computer-based system and method for organizing digital photos in accordance with objects 
detected and recognized in the digital photos. In one implementation described in detail below, a 
group of digital photos is organized by detecting and recognizing faces appearing therein. 
[0017] The present invention may be implemented as a computer-executed software 

product, installed on a user's personal computer (or some other suitable device) to allow the user 
to organize a collection of digital photographs in accordance with the people appearing in the 
photographs. In one embodiment, a system for organizing and processing photographs is 
provided with at least one face recognition algorithm, which determines the similarity of faces 
appearing in a collection of photographs to at least one face in a model folder. The present 
invention may be implemented to allow a user to set up folders for different people, find similar 
faces to a face or faces in these folders, and add photographs to the "people folders" based on 
detected similarity. One implementation of the present invention is described in which face 
detection and recognition are used as tools (e.g., as a software plug-in for a photo 
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storing/processing application) for facilitating digital photo organization (e.g., allowing the user 
to organize hundreds or photographs based on people in a few minutes). Although the following 
description details a specific work-flow with reference to specific display screens shown in the 
figures, it should be recognized that many variations are possible. 

[0018] FIG. 1 is a block diagram of a system for organizing digital photos according to 

an embodiment of the present invention. The system 10 illustrated in FIG. 1 includes the 
following components: an image input device 20; a media organizing and processing unit 30; a 
printing unit 40; a user input unit 50; and a display 60. Operation of and functional interaction 
between the components illustrated in FIG. 1 will become apparent from the following 
description. 

[0019] The image input device 20 provides digital image data representing a photograph. 

The image input device 20 may be one or more of any number of devices for providing digital 
image data derived from photographic film or a digital camera, e.g., a recording medium (a CD- 
R, floppy disk, etc.) or a network connection. The image input device 20 may be a scanner for 
scanning images recorded on paper or film, e.g., including CCD sensors for photoelectronically 
reading R (red), G (green), and B (blue) image information from film, frame by frame. The 
media organizing and processing unit 30 receives digital image data from the image input device 
20 and performs recognition-based media organizing in a manner discussed in detail below. 
Although not a focus of this application, the media organizing and processing unit 30 may 
perform other functions, such as image compression, editing, color/density correction, etc., in 
accordance with commands received from the user input device 50. A user views graphical user 
interface (GUI) display screens and other information (including photos) output by the media 
organizing and processing unit 30 on a display 60 and inputs commands and other information to 
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the media organizing and processing unit 30 via the user input device 50. In the embodiment 
illustrated in FIG. 1, the user input device 50 includes a keyboard 52 and a mouse 54. The 
printing unit 40 selectively receives image data from the media organizing and processing unit 
30 to generate prints. 

[0020] FIG. 2 illustrates details of the media organizing and processing unit 30 for 

performing recognition-based media organizing of digital photos according to an embodiment of 
the present invention. As shown in FIG. 2, the media organizing and processing unit 30 according 
to this embodiment includes: an image memory 32, which stores digital image data; a face 
detection/cropping unit 34; a face recognition unit 36; and a media organizing control unit 38. 
Although the various components of FIG. 2 are illustrated as discrete elements, such an illustration 
is for ease of explanation and it should be recognized that certain operations of the various 
components may be performed by the same physical device, e.g., by a microprocessor of a user's 
personal computer. Operation of the components of the media organizing and processing unit 30 in 
FIG. 2 will next be described with reference to FIGs. 3-7. 

[0021] FIG. 3 is an exemplary GUI display screen 100 output by the media organizing 

control unit 38 to the display 60 in accordance with one implementation of the present invention. 
The GUI display screen 100 illustrated in FIG. 3 allows the user to initiate and control various 
media organizing and viewing functions provided by the media organizing control unit 38. The 
display screen 100 includes a category list 102 of general media organizing categories, including 
"dates," "events," "places," and "people," which constitute digital photo albums. Accordingly, 
the user may select from this category list 102 to organize digital photos and/or access 
previously-organized digital photos based on these general categories. For example, the user 
may select "dates" from the category list 102 to organize and/or browse digital photos organized 
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by date. Likewise, the user may select "events" to organize and/or browse digital photos 
organized by the event at which they were taken (e.g., vacation 2002, graduation 2003, etc.). 
Furthermore, in accordance with principles of the present invention, the user may select 
"people" to organize digital photos based on the faces appearing therein, using face recognition, 
and browse digital photos previously organized based on people. Each general category in the 
category list 102 will typically include multiple folders, which may further contain sub-folders. 
Thus, under the "people" category, separate folders may be set up for "family," "classmates," 
"work colleagues," etc., each containing a sub-folder for an individual member of the particular 
category. 

[0022] The GUI display screen 100 further includes a display window 110 for viewing 

photos. In the example of FIG. 3, the display window 110 displays an index view of photos in a 
selected folder. This may be initiated by selecting a "show all photos" button 1 14. Also, the user 
may view cropped faces appearing in photos of the selected folder in the display window 1 10 by 
selecting a "show faces" button 116. Extracting and cropping faces from a group of digital 
photos will be described in greater detail below. An individual photo may be viewed by clicking 
on the desired photo in the display window 110 and selecting a "photo viewer" tab 112. In 
accordance with principles of the invention, the face recognition-based organization may be 
initiated by selecting a "search faces" button 118 from the display screen 100. The display 
screen 100 includes various additional icons and menus for performing related functions, such as 
ordering prints, sharing photos, editing images, etc. 

[0023] FIG. 4 is a flow diagram illustrating functions performed by the media organizing 

and processing unit 30 to organize digital photos based on face recognition in accordance with 
an embodiment of the present invention. Initially, digital images are imported for organization in 
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one or more of the digital photo album categories in list 102 (step S212). These digital images 
may be imported from existing folders (e.g., stored in image memory 32) or input via external 
media, such as the image input device 20. FIG. 5 illustrates an exemplary GUI display screen 
300 for importing digital photos, which allows the user to select folders or individual images to 
be imported. The face detection/cropping unit 34 detects faces in the imported digital photos and 
crops out the face(s) from the digital photos to obtain a view of the individual faces in the 
imported digital photos (S214). The face detection/cropping unit 34 may utilize a face detection 
algorithm such as that disclosed in U.S. Patent Application 10/440,173, titled "Method and 
Apparatus for Red-Eye Detection," which is herein incorporated by reference. Other techniques 
for face detection may also be utilized. 

[0024] The user is able to control whether to automatically detect and crop out faces for 

images being imported, e.g., by selecting the "automatically detect faces" button 306 on the GUI 
display screen 300 shown in FIG. 5. In other words, the face detection/cropping unit 34 may 
perform face detection and cropping in the background as the user is performing other functions. 
If a user decides not to detect faces at the time of importing the photos, the user can select a 
"detect faces" menu option (not shown) at any time. Again, the face detection/cropping unit 34 
may operate in the background with low priority, meaning that other tasks (operations) can be 
performed while the detection is being done. The detect faces operation need only be performed 
once for each imported digital photo. After face detection and cropping has been performed for a 
set of imported digital photos, the user may select a "show faces" button 1 16 of the GUI display 
screen 100 to view the detected faces in the display window 110. 

[0025] At any time after face detection and cropping for a group of imported digital 

images has been executed, the user may initiate face recognition-based organization of such 
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digital photos. FIG. 6 is an exemplary graphical user interface display screen associated with 
displaying face images sorted by similarity to model face images to assist the user in organizing 
digital photos. Although the face detection/cropping unit 34 extracts faces appearing in digital 
photos, it can be seen from FIG. 6 that the face view may show an area larger than the face (face 
area 414) so that the user can view the entire face and surrounding features, such as the head, 
neck, and hat (if worn). This results in a better viewing image as compared to just viewing the 
face itself. 

[0026] The user may select a folder 402 under the "people" category in GUI display 

screen 400 (in category list 102) or create a new person folder (S216). To serve as a basis for 
face recognition, the selected folder or new folder must contain at least one face image. The user 
may drag and drop any of the faces from an "unknown" display window portion 406 (i.e., 
displaying faces from imported digital photos that have not been associated with a particular 
person folder). Once the user has selected the person folder for matching with "unknown" faces 
(which serves as a comparison model), the face recognition unit 36 applies face recognition to 
determine similarity of unknown faces to the selected model (S218). 

[0027] A previously classified image of a person may appear substantially different than 

the same person in other photographs (e.g., due to different angles, poses, photographing 
positions, aging, etc.). Thus, a preferred embodiment of the present invention utilizes a face 
recognition algorithm that provides good results under such conditions. One implementation of 
the present invention utilizes the face recognition algorithm described in the U.S. Patent 
application titled "Method and Apparatus for Object Recognition Using Probability Models" 
filed on even date herewith and which is hereby incorporated by reference. Such a face 
recognition algorithm has also been described in U.S. Provisional Application No. 60/519,639 
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filed November 14, 2003, which has been incorporated herein by reference in its entirety. After 
the face recognition unit 36 obtains a similarity measure for each unknown face, the media 
organizing control unit 38 sorts the unknown faces by similarity measure (most similar to least 
similar) and outputs an updated "unknown" display window portion 406 (FIG. 6) showing the 
unknown faces in sorted order (S220). Using this updated display 400, the user is able to drag 
and drop one or more of the sorted unknown faces from the "unknown" display window portion 
406 into a model display window portion 408 (step S222). If, after the user has dragged and 
dropped an unknown face into the model display window portion 408, there are additional 
unknown faces (S224), the face recognition algorithm is repeated (S218). In other words, the 
face recognition unit 36 in accordance with one embodiment of the present invention may output 
different results depending on the images in the model folder 402, particularly because as 
additional faces from different conditions are added to the folder 402, the accuracy of the 
recognition improves and the recognition results change. Thus, the arrangement of unknown 
faces in display window portion 406 is updated as the user drags and drops unknown faces into 
the model display window portion 408 to improve the user's ability to quickly and reliably 
associate unknown faces and the corresponding digital photos with the model folder 402. When 
the media organizing control unit 38 determines that the people-based media organizing is 
complete (S224), the process ends. Using the above-described technique, the user is able to 
organize even hundreds of photos in a short time based on the people appearing therein. 
[0028] Having detected and recognized faces in digital photos, the resulting information 

can be used in accordance with principles of the present invention to enhance photo viewing 
presentations. For example, as shown in FIG. 7A, the folder name corresponding to a face in a 
digital photo can be displayed along with the digital photo. This may be done during a "slide 
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presentation" of a group of digital photos. Furthermore, as shown in FIG. 7B, the name of a 
classified face may be displayed when the user moves the cursor over a displayed face. 
Furthermore, by clicking on the person's name, the media organizing control unit 38 may 
initiate a zooming function to zoom in on the detected person as shown in FIGs. 7B-7C. More 
than one person can be specified to initiate a panning display between multiple faces in the 
digital photo being viewed. 

[0029] Although embodiments of the present invention have been described above in the 

context of face-recognition, principles of the present invention may be applied to organizing 
digital photos based on other types of objects that can be detected and recognized in the digital 
photos. The invention having thus been described, it should be apparent that various other 
modifications are possible without departing from the spirit and scope of the present invention. 
For example, depending on accuracy, digital photos may be automatically grouped in people 
folders based on face recognition, providing the user with the option of re-assigning incorrectly 
grouped photos. 
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